Method for setting a viewing direction in a representation of a virtual environment

ABSTRACT

A method for setting a direction of view in a representation of a virtual environment is disclosed. The method includes recording a known object in a real environment using a recording device. Further, the method includes determining a rotational offset of the direction of view in the representation of the virtual environment around a yaw axis of the representation of the virtual environment based on the recording of the object, a known position of the recording device in the real environment and a current direction of view in the representation of the virtual environment. The method further includes rotating the direction of view in the representation of the virtual environment by the rotational offset.

TECHNICAL FIELD

Embodiments relate to the representation of a virtual environment. Inparticular, embodiments relate to a method for setting a direction ofview in a representation of a virtual environment.

BACKGROUND

The representation and simultaneous perception of a computer-generatedinteractive virtual environment and its physical characteristics iscalled virtual reality (VR). To generate a feeling of immersion, arepresentation of the virtual environment may be transferred to a usere.g. via a display device mounted to the head of the user. Such devicesare known as Head-Mounted Display (HMD), Head-Mounted Display Unit orHead-Mounted Unit (HMU). The display device represents the virtualenvironment e.g. on a near-to-eye display or projects it directly to theretina of the user. The orientation, i.e. the direction of view, in therepresentation of the virtual environment is here set by rotating arounda transverse axis (pitch axis) of the representation of the virtualenvironment, rotating around a longitudinal axis (roll axis) of therepresentation of the virtual environment and/or rotating around avertical axis (yaw axis, normal axis) of the representation of thevirtual environment. The pitch axis, the roll axis and the yaw axis hereare perpendicular to one another.

To adapt the representation of the virtual environment to movements ofthe user, i.e. to navigate through the virtual environment according tothe movements of the user, the position of the head of the user may bedetected. For example, a position and an orientation of the head of theuser in the real environment, i.e. the real world, may be determined toadapt the representation of the virtual environment. Accordingly, theperception of one's own person in the real environment may be reducedand identification with the virtual environment be increased. In orderto detect the user's head, e.g. the propagation times of a radio signalfrom one single transmitter at the head of the user to several remotereceivers may be used. This way, e.g. by means of aTime-Difference-of-Arrival (TDoA) method, from the differentdifferential times between transmitting the radio signal by thetransmitter and receiving the radio signal by the respective receiver, aposition of the head of the user may be determined at an accuracy in thesingle-digit centimeter range. The transmitter may here, e.g., beintegrated in a display device mounted to the head of the user or beattached to the head of the user independent of the display devicemounted to the head of the user. A user may thus change the representedposition in the virtual environment e.g. by freely moving around in thereal environment. Alternatively, e.g. via a camera-based method, a timeof flight (ToF) method, a round trip time (RTT) method and/or aninertial measurement unit (IMU), the position of the head of the usermay be detected.

The orientation of the head of the user may, for example, be determinedby a corresponding sensors (e.g. gyroscope, magnetometer, accelerometer)of the display device mounted to the head of the user. When the displaydevice mounted to the head of the user for example comprises a mobilecommunications device like, e.g., a smartphone and a fixing device formounting the mobile communications device to the head of the user,sensors already present in the mobile communications device may beutilized for determining the orientation of the head of the user. A usermay thus change the direction of view in the virtual environment e.g. byrotating or tilting the head in the real environment. When rotating thehead in the real environment e.g. the direction of view in the virtualenvironment is changed by rotating around the yaw axis of therepresentation of the virtual environment. For example, the magnetometermay be used to determine the orientation in the real environment in asufficiently stable way. For a self-contained area in the realenvironment e.g. a magnetic field map may be generated so that using anaccordingly calibrated magnetometer the orientation of the head of theuser in the real environment may be determined.

The determination of the orientation of the head of the user by means ofthe above mentioned sensors may lead to orientation errors, however.Thus, magnetometers may also provide wrong measurement values, so thatthe measured orientation of the head does not correspond to the realorientation of the head in the real environment. Also an approximatedetermination of the orientation of the head by coupling the measurementvalues of a gyroscope and an accelerometer may lead to a discrepancybetween the measured and/or determined orientation of the head and thereal orientation of the head in the real environment due to measurementerrors of the individual sensor elements. Thus, for example, thecombination and integration of erroneous measurement values over alonger period of time may lead to deviations between the determinedorientation of the head and the real orientation of the head in the realenvironment. Also frequent and intensive changes of the rotation rate ofthe sensors (e.g. changing between slow and fast rotation movements ofthe head) may lead to significant deviations between the determinedorientation of the head and the real orientation of the head in the realenvironment, wherein the error increases with an increase of therotation rate changes. Accordingly, also the orientation of therepresentation of the virtual environment which is based on themeasurement values is corrupted.

A rotational offset of the direction of view in the representation ofthe virtual environment around the yaw axis of the representation of thevirtual environment up to approx. ±15° is usually not perceived by auser. If a user is e.g. walking straight ahead in the real environment,he may not realize when the direction of view in the representation ofthe virtual environment deviates by up to approx. ±15° (i.e. thedirection of view is rotated by up to 15° to the left and/or rightaround the yaw axis). In other words: Up to a certain degree the userdoes not note that in contrast to the real environment he is not movingstraight ahead in the virtual environment but at an angle. Largerdeviations are noted by the user, however, and reduce the feeling ofimmersion. Due to the measurement errors of the sensors such anundesirably large rotational offset of the direction of view may resultin the representation of the virtual environment. In particular when theuser utilizes the representation of the virtual environment over alonger period, due to the integration of the erroneous measurementvalues a substantial offset of the direction of view may result in therepresentation of the virtual environment. The notable deviation of thedirection of view in the representation of the virtual environment mayalso lead to a discomfort of the user.

There is thus a demand of providing a possibility for correcting thedirection of view in the representation of the virtual environment.

SUMMARY

This object is solved by embodiments of a method for setting a directionof view in a representation of a virtual environment. Here, the methodcomprises recording a known object in a real environment using arecording device (e.g. an image, video or sound recording). Further, themethod comprises determining a rotational offset of the direction ofview in the representation of the virtual environment around a yaw axisof the representation of the virtual environment based on the recordingof the object, a known position of the recording device in the realenvironment and a current direction of view in the representation of thevirtual environment. The method further comprises rotating the directionof view in the representation of the virtual environment by therotational offset.

The recording device may spatially be located in close proximity to theuser. For example, the recording device may be mounted to the body ofthe user (like e.g. the head). From the recording of the object and theknown position of the recording device in the real environment theorientation of the recording device in the real environment may bedetermined, which may then approximately be assumed to be theorientation of the head of the user in the real environment. Therefrom,using the information on the current direction of view in therepresentation of the virtual environment, the rotational offset of thedirection of view in the representation of the virtual environmentaround the yaw axis of the representation of the virtual environment maybe determined and the representation of the virtual environment may becorrected accordingly. The representation of the virtual environment maythus be adapted to the actual position and orientation of the head ofthe user in the real environment. Among other things, embodiments of theproposed method thus allow a calibration of the direction of view in therepresentation of the virtual environment. An improved feeling ofimmersion may be generated for the user.

Further embodiments relate to a second method for setting a direction ofview in a representation of a virtual environment. Here, the methodcomprises recording a known object in a real environment using arecording device (e.g. an image, video or sound recording). Further, themethod comprises determining a rotational offset of the direction ofview in the representation of the virtual environment around a yaw axisof the representation of the virtual environment based on the recordingof the object and a current direction of view in the representation ofthe virtual environment. Additionally, the method comprises rotating thedirection of view in the representation of the virtual environment bythe rotational offset.

From the recording of the object the orientation of the recording devicein the real environment may be determined. If the recording device isspatially located in close proximity to the user (e.g. at the head ofthe user), the orientation of the recording device in the realenvironment may approximately be assumed to be the orientation of thehead of the user in the real environment. Therefrom, along with theinformation on the current direction of view in the representation ofthe virtual environment, the rotational offset of the direction of viewin the representation of the virtual environment around the yaw axis ofthe representation of the virtual environment may be determined and therepresentation of the virtual environment may be corrected accordingly.The representation of the virtual environment may thus be adapted to theactual position and orientation of the head of the user in the realenvironment. Among other things, embodiments of the proposed method thusallow a calibration of the direction of view in the representation ofthe virtual environment. An improved feeling of immersion may begenerated for the user.

Further embodiments relate to a third method for setting a direction ofview in a representation of a virtual environment. Here, the methodcomprises recording an object arranged in a real environment at the bodyof the user using a recording device arranged at the head of the user ata first time instant and at a later second time instant (e.g. an image,a video or a sound recording). Further, the method comprises determininga rotational offset of the direction of view in the representation ofthe virtual environment around a yaw axis of the representation of thevirtual environment based on the recordings of the object at the firsttime instant and at the second time instant and measurement values of atleast one further sensor mounted to the head of the user. The methodfurther comprises rotating the direction of view in the representationof the virtual environment by the rotational offset.

From the recordings of the object at the first time instant and at thesecond time instant and the measurement values of at least one furthersensor mounted to the head of the user each an effective rotation of therecording device around the yaw axis of the head of the user between thefirst time instant and the second time instant may be determined. Thedifference between the two determined values for the rotation of therecording device around the yaw axis of the head of the user mayapproximately be assumed to be the rotational offset of the direction ofview in the representation of the virtual environment around the yawaxis of the representation of the virtual environment. Therepresentation of the virtual environment may thus be correctedaccordingly. The representation of the virtual environment may thus beadapted to the actual position and orientation of the head of the userin the real environment. Among other things, embodiments of the proposedmethod thus allow a calibration of the direction of view in therepresentation of the virtual environment. Thus, an improved feeling ofimmersion may be generated for the user.

In a further aspect, embodiments comprise a program having a programcode for executing at least one of the proposed methods, when theprogram code is executed on a computer, a processor, or a programmablehardware component.

BRIEF DESCRIPTION OF THE FIGURES

Embodiments are explained in more detail with reference to theaccompanying Figures, in which:

FIG. 1 shows an example of a method for setting a direction of view in arepresentation of a virtual environment;

FIG. 2 shows an example of a connection between an object in the realenvironment and a recording of the object;

FIG. 3a shows a first example for an object;

FIG. 3b shows a second example for an object;

FIG. 3c shows a third example for an object;

FIG. 3d shows a fourth example for an object;

FIG. 4 shows exemplary features in an object;

FIG. 5 shows an example of an allocation of the positions of features ofan object in the real environment to the position of the respectivefeature in a recording of the object;

FIG. 6 shows an example of a histogram of certain orientations of therecording device in the real environment;

FIG. 7 shows a sequence of recordings of the same object rotated withrespect to each other;

FIG. 8 shows a further example of a method for setting a direction ofview in a representation of a virtual environment;

FIG. 9a shows a recording of a fifth example of an object;

FIG. 9b shows a binary recording corresponding to the recordingillustrated in FIG. 9 a;

FIG. 9c shows a recording of a sixth example of an object;

FIG. 9d shows a binary recording corresponding to the recordingillustrated in FIG. 9 b;

FIG. 10 shows a further example of a method for setting a direction ofview in a representation of a virtual environment;

FIG. 11a shows an example of a connection between a motion vector of auser, an actual direction of view of the user and a direction of viewdetermined from measurement values of at least one further sensorarranged at the head of the user in the representation of the virtualenvironment at a first time instant; and

FIG. 11b shows an example of a connection between a motion vector of auser, an actual direction of view of the user and a direction of viewdetermined from measurement values of at least one further sensorarranged at the head of the user in the representation of the virtualenvironment at a second time instant.

DESCRIPTION

Various embodiments will now be described with reference to theaccompanying drawings in which some example embodiments are illustrated.In the Figures, the thicknesses of lines, layers and/or regions may beexaggerated for clarity.

Like numbers refer to like or similar components throughout thefollowing description of the included figures, which merely show someexemplary embodiments. Moreover, summarizing reference signs will beused for components and objects which occur several times in oneembodiment or in one Figure but are described at the same time withrespect to one or several features. Components and objects describedwith like or summarizing reference signs may be implemented alike oralso differently, if applicable, with respect to one or more or all thefeatures, e.g. their dimensioning, unless explicitly or implicitlystated otherwise in the description.

Although embodiments may be modified and changed in different ways,embodiments are illustrated as examples in the Figures and are describedherein in detail. It is to be noted, however, that it is not intended torestrict embodiments to the respectively disclosed forms but thatembodiments rather ought to c any functional and/or structuralmodifications, equivalents and alternatives which are within the scopeof the invention. Same reference numerals designate same or similarelements throughout the complete description of the figures.

It is noted, that an element which is referred to a being “connected” or“coupled” to another element, may be directly connected or coupled tothe other element or that intervening elements may be present. If anelement is referred to as being “directly connected” or “directlycoupled” to another element, no intervening elements are be present.Other terms used to describe a relationship between elements ought to beinterpreted likewise (e.g. “between” versus “directly between”,“adjacent” versus “directly adjacent”, etc.).

The terminology used herein only serves for the description of specificembodiments and should not limit the embodiments. As used herein, thesingular form such as “a,” “an” and “the” also include the plural forms,as long as the context does not indicate otherwise. It will be furtherunderstood that the terms e.g. “comprises,” “comprising,” “includes”and/or “including,” as used herein, specify the presence of the statedfeatures, integers, steps, operations, elements and/or components, butdo not preclude the presence or addition of one and/or more otherfeatures, integers, steps, operations, elements, components and/or anygroup thereof.

Unless otherwise defined, all terms (including technical and scientificterms) are used herein in their ordinary meaning of the art to which theexamples belong and given to same by a person of ordinary skill in theart. It is further clarified that terms like e.g. those defined iongenerally used dictionaries are to be interpreted to have the meaningconsistent with the meaning in the context of relevant technology, aslong as it is not expressly defined otherwise herein.

FIG. 1 shows a method 100 of setting a direction of view in arepresentation of a virtual environment. The virtual environment is acomputer-generated interactive world with predetermined physicalcharacteristics which are e.g. output to a user. The direction of viewin the representation of the virtual environment here corresponds to theorientation (alignment) of the section of the virtual environmentillustrated in the representation of the virtual environment in thevirtual environment. The representation of the virtual environment may,for example, represent the virtual environment from a first-person viewor first-person perspective, i.e. the virtual environment may berepresented or reproduced as a user would see it if he or she actuallymoved around in the virtual environment. Accordingly, the direction ofview in the representation of the virtual environment would correspondto the direction of view of the user if he or she actually moved aroundin the virtual environment. The direction of view in the representationof the virtual environment is here set by rotating around the transverseaxis of the representation of the virtual environment, rotating aroundthe longitudinal axis of the representation of the virtual environmentand/or rotating around the yaw axis of the representation of the virtualenvironment.

Here, the method 100 comprises recording 102 a known object in a realenvironment (i.e. the real world) using a recording device. The knownobject may both be an object in the real environment especially placedfor the proposed method and also an object already existing in the realenvironment. For example, the object may be placed in an area of thereal environment in which the user moves around or an already existingelement of this area. The object may both be a basically two-dimensional(planar) object, i.e. an object basically extending only in two spatialdirections, and also a three-dimensional object, i.e. an objectextending in similar dimensions or orders of magnitude in all threespatial directions. If the user moves in the real environment, e.g.within a room or a hall, the known object may e.g. be an object of theroom and/or hall, like e.g. a window, an illumination device, a door, apost, a carrier, a piece of furniture or another element of the roomand/or the hall. Alternatively, the object may e.g. be a poster, aprojection, a sound source or another element which has especially beenplaced in the room and/or hall for the proposed method.

The recording may e.g. be a still picture (i.e. a single recording), avideo (i.e. a sequence of images) or a sound recording, i.e. a recordingof sound (e.g. sounds, noises, music or speech). Accordingly, therecording device may comprise a still camera, a video camera, a (stereo)sound recording device or a combination thereof.

Further, the method 100 comprises determining 104 a rotational offset ofthe direction of view in the representation of the virtual environmentaround the yaw axis of the representation of the virtual environmentbased on the recording of the object, a known position of the recordingdevice in the real environment and a current direction of view in therepresentation of the virtual environment. The recording device mayspatially be arranged in close proximity to the user. A position of theuser in the real environment measured during regular operation of a VRsystem may then e.g. be used as the position of the recording device. Ifthe position of the user is determined e.g. by a time of flightmeasurement, as described hereinabove, the transmitter may e.g. bearranged at the head of the user and the recording device in spatialproximity to the same to detect the position of the recording device inthe real environment as exactly as possible. The current direction ofview in the representation of the virtual environment may, for example,be received by a display device (e.g. HMD, HMU) which outputs therepresentation of the virtual environment to the user (and optionallycalculates the same) or by a computer which calculates the virtualenvironment (e.g. back end of a VR system).

From the recording of the object and the known position of the recordingdevice in the real environment the orientation of the recording devicein the real environment may be determined, which may approximately beassumed to be the orientation of the user or his/her head in the realenvironment. Therefrom, using the information on the current directionof view in the representation of the virtual environment, the rotationaloffset of the direction of view in the representation of the virtualenvironment around the yaw axis of the representation of the virtualenvironment may be determined. Some examples for the determination ofthe orientation of the recording device in the real environment and alsofor the determination of the rotational offset of the direction of viewin the representation of the virtual environment are explained in moredetail in the following description.

The method 100 further comprises rotating 106 the direction of view inthe representation of the virtual environment by the rotational offset.In other words: The direction of view in the representation of thevirtual environment is corrected by a rotation around the yaw axis ofthe representation of the virtual environment, wherein the direction andthe magnitude of the rotation are determined by the rotational offset.The representation of the virtual environment is thus corrected by therotational offset. The representation of the virtual environment maythus be adapted to the actual position and orientation of the head ofthe user in the real environment. Among other things, the method 100thus allows a calibration of the direction of view in the representationof the virtual environment. In particular, using the method 100 anerroneously determined orientation in the real environment and/or adrifting of the direction of view in the representation of the virtualenvironment caused by measurement errors of the conventionally usedsensors for determining the position and the alignment (of the head) ofa user may be corrected.

As already indicated above, the method may in some embodiments furthercomprise outputting the representation of the virtual environment to auser. Outputting the representation of the virtual environment to theuser may here e.g. be executed via a display device mounted to the headof the user which further comprises the recording device. In such anarrangement the orientation of the recording device in the realenvironment may approximately be assumed to be the orientation of thehead of the user in the real environment.

In some embodiments, the display device mounted to the head of the usercomprises a mobile communications device (e.g. a smartphone). Asindicated above, in a conventional operation of the VR system sensorsalready existing in the mobile communications device (e.g. gyroscope,magnetometer, accelerometer) may be used for determining the orientationof the head of the user in the real environment. By using the camera ofthe mobile communications device as a recording device, a rotationaloffset of the direction of view in the representation of the virtualenvironment around the yaw axis of the representation of the virtualenvironment due to measurement errors of the sensors of the mobilecommunications device may be corrected. For the calibration of therepresentation of the virtual environment, thus resources may be usedalready provided by the mobile communications device. In other words:The method 100 may be executed directly (i.e. online) on the mobilecommunications device. The method 100 may thus enable a calibration ofthe representation of the virtual environment without additionalhardware components. Alternatively, e.g. a part of the method 100 may beexecuted by the mobile communications device and another part of themethod 100, like e.g. determining 104 the rotational offset of thedirection of view in the representation of the virtual environmentaround the yaw axis of the representation of the virtual environment maybe executed by an already existing back end of the VR system used by theuser (i.e. offline). The determined rotational offset may then e.g. betransmitted from the back-end to the mobile communications device sothat the same may rotate the direction of view in the currentrepresentation of the virtual environment by the rotational offset. Theabove described functionality may e.g. be implemented by an update forone or several already existing software components of the VR system(e.g. software for the mobile communications device or software for theback end).

In some embodiments, determining 104 the rotational offset of thedirection of view in the representation of the virtual environmentaround the yaw axis of the representation of the virtual environment maycomprise determining an orientation of the recording device in the realenvironment based on the recording of the object and the known positionof the recording device in the real environment (exemplary methods willbe discussed in the following). Further, determining 104 the rotationaloffset may comprise determining a target direction of view in therepresentation of the virtual environment based on the orientation ofthe recording device in the real environment. For example, thedetermined orientation of the recording device in the real environmentmay be provided to an algorithm for the calculation of therepresentation of the virtual environment based which calculates arepresentation of the virtual environment based thereon. In particularwhen the recording device is arranged at the head of the user, thetarget direction of view in the virtual environment may be the directionof view in the virtual environment which corresponds to the actualposition and orientation of the head of the user in the realenvironment. For example, the recording device may be aligned straightahead in the direction of view of the user in the real environment orvertically to the same. The direction of view in the calculatedrepresentation of the virtual environment may consequently be regardedas the target direction of view.

From the target direction of view in the representation of the virtualenvironment and the current direction of view in the representation ofthe virtual environment, according to embodiments, the rotational offsetof the direction of view in the representation of the virtualenvironment around the yaw axis of the representation of the virtualenvironment is determined. This may, for example, be done by acomparison of the target direction of view and the current direction ofview in the representation of the virtual environment. In other words:It is determined to what extent the current direction of view in therepresentation of the virtual environment is rotated relative to thetarget direction of view in the representation of the virtualenvironment around the yaw axis of the representation of the virtualenvironment.

The representation of the virtual environment may e.g. be rendered bythe display device (e.g. including a mobile communications device)mounted to the head of the user. The determination of the targetdirection of view for a time instant t₀ may e.g. be executed by a backend of the VR system and subsequently be transmitted to the mobilecommunications device mounted to the head of the user. From the targetdirection of view in the representation of the virtual environment forthe time instant t₀ and the current direction of view in therepresentation of the virtual environment at the time instant t₀ themobile communications device may then determine the rotational offset ofthe direction of view around the yaw axis of the representation of thevirtual environment at the time instant t₀. Assuming that a furtherdrift of the direction of view in the representation of the virtualenvironment between the time instant t₀ and a later time instant t₁ maybe neglected, the mobile communications device may e.g. rotate thedirection of view in the representation of the virtual environment forthe time instant t₁ by the rotational offset of the direction of view atthe time instant t₀, i.e. correct the same. Accordingly, therepresentation of the virtual environment may be output to the user witha correct direction of view.

The above-described method may be executed repeatedly during theutilization of the VR system. Thus e.g. a further drift of the directionof the direction of view in the representation of the virtualenvironment between the time instant t0 and at the later time instant t₁may be corrected. Further, after a correction has been executed, themethod may at least partially be executed again to verify the precedingcorrection.

In FIG. 2 it is illustrated exemplarily how the orientation of arecording device in the real environment may be determined based on arecording 210 of a known object 220 and a known position C of therecording device in the real environment. The object 220 may be regardedas an amount of world points M. The recording 210 may be regarded as anamount of image points m.

The orientation of the recording device and thus the angle of view inthe recording 210 may generally be determined from a transformationwhich transfers the world points M of the object 220 into correspondingimage points m of the recording 210. In general, the transformation maybe represented as follows:

m=KR[I|−C]M  (1),

wherein I represents the unity matrix and K and R the breakdown of thecamera matrix of the recording device, wherein K represents theintrinsic matrix describing the focal length, the main point of thecamera and the deviations of the axes of the image coordinate systemfrom the assumed orthogonality (axis skew), and R represents a generalrotational matrix. R may here be represented as a product of threerotational matrices R_(x), R_(y) and R_(z) around unity directions X, Yand Z which are orthogonal to each other. From a defined origin in thereal environment e.g. X may point to the right, Y upward (i.e.heavenward) and Z into the depth (i.e. to the front). The unitydirection Y thus corresponds to the yaw axis (vertical axis), i.e. arotation around this axis horizontally shifts a recording. The axes ofthe coordinate system of the virtual environment may be selecteddifferent to the orthogonally arranged unity directions X, Y and Z. Oneposition in the real environment may then be translated into a positionin the virtual environment via a coordinate transformation. Accordingly,equation (1) may be transformed as follows:

m=KR _(x) R _(y) R _(z)[I|−C]M  (2)

R _(z) ⁻¹ K ⁻¹ m=R _(x) R _(y)[I|−C]M  (3)

The three rotational matrices R_(x), R_(y) and R_(z) may here depend onan angle α, which indicates the desired horizontal alignment of therecording device in the real environment. In other words: The angle αdefines an orientation (alignment) of the recording device in the planedspanned by X and Z. The rotational matrices R_(x), R_(y) and R_(z) aredefined as usual:

$\begin{matrix}{{R_{x}(\alpha)} = \begin{pmatrix}1 & 0 & 0 \\0 & {\cos \mspace{14mu} \alpha} & {{- \sin}\mspace{14mu} \alpha} \\0 & {\sin \mspace{14mu} \alpha} & {\cos \mspace{14mu} \alpha}\end{pmatrix}} & (4) \\{{R_{y}(\alpha)} = \begin{pmatrix}{\cos \mspace{14mu} \alpha} & 0 & {\sin \mspace{14mu} \alpha} \\0 & 1 & 0 \\{{- \sin}\mspace{14mu} \alpha} & 0 & {\cos \mspace{14mu} \alpha}\end{pmatrix}} & (5) \\{{R_{z}(\alpha)} = \begin{pmatrix}{\cos \mspace{14mu} \alpha} & {{- \sin}\mspace{14mu} \alpha} & 0 \\{\sin \mspace{14mu} \alpha} & {\cos \mspace{14mu} \alpha} & 0 \\0 & 0 & 1\end{pmatrix}} & (6)\end{matrix}$

Assuming R_(x)(α), R_(z)(α), K and C are known, R_(y)(α) may bedetermined from a corresponding pair M↔m. R_(x)(α) and R_(z)(α) may e.g.be determined using the sensors already existing in the mobilecommunications device (e.g. via the gravitational vector). Here, thecoefficients of equation (3) may be summarized as follows:

C _(q) =R _(z) ⁻¹ K ⁻¹  (7)

C _(p) =R _(x)[I−C]  (8)

Accordingly, equation (3) may be represented as follows:

C _(q) m=R _(y) C _(p) M  (9)

and/or

Y=R _(y) X  (10),

wherein Y=C_(q)m and X=C_(p)M.

Accordingly, equation (10) may be transformed as follows:

$\begin{matrix}{\begin{pmatrix}u^{\prime} \\v^{\prime} \\1\end{pmatrix} = {\begin{pmatrix}{\cos \mspace{14mu} \alpha} & 0 & {\sin \mspace{14mu} \alpha} \\0 & 1 & 0 \\{{- \sin}\mspace{14mu} \alpha} & 0 & {\cos \mspace{14mu} \alpha}\end{pmatrix}\begin{pmatrix}u \\v \\1\end{pmatrix}}} & (11)\end{matrix}$

Multiplying equation (11) results in the following equation system:

$\begin{matrix}{u^{\prime} = \frac{{{u \cdot \cos}\mspace{14mu} \alpha} + {\sin \mspace{14mu} \alpha}}{{\cos \mspace{14mu} \alpha} - {{u \cdot \sin}\mspace{14mu} \alpha}}} & (12) \\{v^{\prime} = \frac{v}{{\cos \mspace{14mu} \alpha} - {{u \cdot \sin}\mspace{14mu} \alpha}}} & (13) \\{1 = 1} & (14)\end{matrix}$

This equation system may be solved for the angle α as follows:

u′·(cos α−u·sin α)=u·cos α+sin α  (15)

u′·cos α−u′·u·sin α=u·cos α+sin α  (16)

u′·cos α−u·cos α−u′·u·sin α−sin α=0  (17)

cos α·(u′−u)−sin α·(u′·u+1)=0  (18)

sin α·(−u′·u−1)+cos α·(u′−u)=0  (19)

With (−u′·u−1)=a and (u′−u)=b the following results:

$\begin{matrix}{\mspace{79mu} {{{{a \cdot \sin}\mspace{14mu} \alpha} + {{b \cdot \cos}\mspace{14mu} \alpha}} = 0}} & (20) \\{{{{a \cdot \sin}\mspace{14mu} \alpha} + {{b \cdot \cos}\mspace{14mu} \alpha}} = \left\{ \begin{matrix}{{\sqrt{a^{2} + b^{2}}{\sin \left( {\alpha + {\tan^{- 1}\left( \frac{b}{a} \right)}} \right)}\mspace{14mu} f\overset{¨}{u}r\mspace{14mu} {alle}\mspace{14mu} a} > 0} \\{{\sqrt{a^{2} + b^{2}}{\cos \left( {\alpha - {\tan^{- 1}\left( \frac{a}{b} \right)}} \right)}\mspace{14mu} f\overset{¨}{u}r\mspace{14mu} {alle}\mspace{14mu} a} < 0}\end{matrix} \right.} & (21)\end{matrix}$

From equation (21) it follows:

$\begin{matrix}{\mspace{20mu} {{{{a \cdot \sin}\mspace{14mu} \alpha} + {{b \cdot \cos}\mspace{14mu} \alpha}} = {{\sin \left( {\alpha + {\tan^{- 1}\left( \frac{b}{a} \right)}} \right)} = 0}}} & (22) \\{\mspace{79mu} {{{{a \cdot \sin}\mspace{14mu} \alpha} + {{b \cdot \cos}\mspace{14mu} \alpha}} = {{\alpha + {\tan^{- 1}\left( \frac{b}{a} \right)}} = 0}}} & (23) \\{\alpha = {{- {\tan^{- 1}\left( \frac{b}{a} \right)}} = {{{- {\tan^{- 1}\left( \frac{u^{\prime} - u}{{{- u^{\prime}} \cdot u} - 1} \right)}}\mspace{14mu} f\overset{¨}{u}r\mspace{14mu} a} = {{{{- u^{\prime}} \cdot u} - 1} > 0}}}} & (24) \\{\alpha = {{- {\tan^{- 1}\left( {- \frac{b}{a}} \right)}} = {{{- {\tan^{- 1}\left( {- \frac{u^{\prime} - u}{{{- u^{\prime}} \cdot u} - 1}} \right)}}\mspace{14mu} f\overset{¨}{u}r\mspace{14mu} a} = {{{u^{\prime} \cdot u} - 1} > 0}}}} & (25) \\{\mspace{79mu} {{\cos \left( {\alpha - {\tan^{- 1}\left( \frac{a}{b} \right)}} \right)} = 0}} & (26) \\{\mspace{79mu} {{\sin \left( {\alpha - {\tan^{- 1}\left( \frac{a}{b} \right)} + \frac{\pi}{2}} \right)} = 0}} & (27) \\{\alpha = {{{- {\tan^{- 1}\left( \frac{a}{b} \right)}} + \frac{\pi}{2}} = {{{\tan^{- 1}\left( \frac{{{- u^{\prime}} \cdot u} - 1}{u^{\prime} - u} \right)}\mspace{11mu} + {\frac{\pi}{2}\mspace{20mu} f\overset{¨}{u}r\mspace{14mu} b}} = {{u^{\prime} - u} > 0}}}} & (28) \\{\alpha = {{{- {\tan^{- 1}\left( {- \frac{a}{b}} \right)}} - \frac{\pi}{2}} = {{{- {\tan^{- 1}\left( \frac{{u^{\prime} \cdot u} + 1}{u^{\prime} - u} \right)}}\mspace{11mu} - {\frac{\pi}{2}\mspace{20mu} f\overset{¨}{u}r\mspace{14mu} b}} = {{u^{\prime} - u} > 0}}}} & (29)\end{matrix}$

u′ and u may be determined for each corresponding pair M↔m so that theangle α may be determined for each pair.

In FIGS. 3a to 3d some examples for possible objects are illustrated inthe following. Here, FIG. 3a shows an amorphous pattern 310 withdifferent grayscales, FIG. 3b a pattern 320 with circles of differentsizes and grayscales, FIG. 3c an illustration 330 of trees and FIG. 3d acollage 340 of strings (e.g. words or numbers). As shown in FIGS. 3a to3d , the object may be manifold. The patterns illustrated in FIGS. 3a to3d may e.g. be applied to a vertical plane in the real environment (e.g.by means of a poster or by means of projection). For example, thepatterns illustrated in FIGS. 3a to 3d may be illustrated at a side wallof a room or a hall in the form of a poster or as a projection. Theobject is not restricted to the examples of FIGS. 3a to 3d , however. Inthe following description further examples for possible objects areillustrated.

According to embodiments, determining the orientation of the recordingdevice in the real environment comprises determining a transformationwhich correlates the at least one part of the recording of the knownobject with at least one part of a comparison recording. The comparisonrecording may provide information on the position of the known object inthe real environment. For example, a database with comparison recordingsmay be held available which show different objects and/or an object fromdifferent perspectives. Here, for each comparison recording additionalinformation on the position of the illustrated object in the real worldare stored (held available). This information which corresponds to theworld points M in the example shown in FIG. 2 may be used to determinethe orientation of the recording device in the real environmentaccording to the example shown in FIG. 2. In the following, FIGS. 4 to 7exemplarily show two different approaches for determining theorientation of the recording device in the real environment on the basisof determining a transformation which correlates the at least one partof the recording of the known object with at least one part of acomparison recording.

For explaining the first approach, in FIG. 4 a pattern with circles ofdifferent sizes and grayscales is illustrated in the form of a poster400 attached to a wall of the real environment as an example for anobject. The pattern comprises a plurality of features 410-1, 410-2, . .. , 410-n. The features 410-1, 410-2, . . . , 410-n may be determined byfeature extraction methods. Examples for feature extraction methods aree.g. the Scale-Invariant Feature Transform (SIFT) algorithm, the SpeededUp Robust Features (SURF) algorithm or the Binary Robust IndependentElementary Features (BRIEF) algorithm. These features may be stored in adatabase as comparison features, i.e. the plurality of comparisonfeatures of the database may comprise different features of the object.Likewise, the comparison features in the database may come fromdifferent recordings of the same object, as the features may bedifferent depending on the direction of view in the comparisonrecording. Among others, features are only detected by the featureextraction method with a certain angle of view of the recording. If, incase of the exemplary poster 400 which comprises the pattern, thepositions of the four corners 401, 402, 403, 404 of the poster 400 areknown in the real environment, also to the individual comparisonfeatures of the database each a position in the real environment may beassociated. In FIG. 4 to each of the corners 401, 402, 403, 404 of theposter 400 an exemplary three-dimensional coordinate related to the realenvironment is associated (e.g. X=16,58, Y=3,19, Z=30,35 for the corner401) so that also for each feature 410-1, 410-2, . . . , 410-n aposition each in the real environment may be determined. Accordingly,each of the comparison features with its associated position may bestored in the database.

These comparison features may now be utilized for determining theorientation of the recording device in the real environment based on therecording of an object and the known position of the recording device inthe real environment. Using the recording device, first of all arecording of the object is made from the known position—in the exampleshown in FIG. 4 a recording of the poster 400 with the pattern. Forexample, the recording device is part of a display device for outputtingthe representation of the virtual environment mounted to the head of theuser, so that by measuring the position of the head of the user duringoperation of the VR system also the position of the recording device is(basically) approximately known.

In the recording at least one feature of the object is detected. In thisrespect, a feature extraction method is applied to the recording (e.g.one of the above-mentioned algorithms). In the recording of the objectitself a position of the feature is further determined. Consequently,the coordinates of the feature in the coordinate system of the recordingare determined. Furthermore, a comparison feature from the plurality ofcomparison features of the database is identified which corresponds tothe feature of the object in the recording. As indicated above, arespective position in the real environment is associated with each ofthe plurality of comparison features. For identification purposes, e.g.known image registration methods may be used. To identify the comparisonfeature from the plurality of comparison features of the database asfast and efficiently as possible, e.g. a nearest neighbor method may beused receiving the position of the feature in the recording of theobject, the position in the real environment of the plurality ofcomparison features and the known position of the recording device inthe real environment as input variables.

From the known position of the recording device in the real environment,the position of the feature in the recording and the position in thereal environment associated with the identified comparison feature, nowthe orientation of the recording device in the real environment may bedetermined according to the principles illustrated in FIG. 2. Relatingto the method illustrated in FIG. 2 for determining the orientation ofthe recording device in the real environment, the known position fromwhich the poster 400 is recorded corresponds to the known position C ofthe recording device in the real environment. The position of thefeature in the recording corresponds to an image point m and theposition in the real environment associated with the identifiedcomparison feature to a world point M. Thus, according to the principlesillustrated in FIG. 2, a transformation may be determined whichcorrelates the position of the feature in the recording to the positionin the real environment associated with the identified comparisonfeature. Accordingly, the orientation of the recording device in thereal environment may be determined.

As may already be seen from FIG. 4, in one recording of an objectseveral features may be detected. In other words: When determining theorientation of the recording device in the real environment severalfeatures of the object may be detected. Accordingly, for the pluralityof detected features of the object a plurality of comparison featuresfrom the database may be identified. An orientation of the recordingdevice in the real environment may be determined for each of the severaldetected features of the object.

An exemplary allocation 500 of features of the object in the recordingto comparison features is depicted in FIG. 5. In FIG. 5 each theposition of a feature of the object (image point) detected in therecording of the object and the position in the real environment (worldpoint) allocated to the respective identified comparison feature areillustrated. The positions are here each given in arbitrary units.Corresponding image points and world points are connected with astraight line in FIG. 5. As indicated in FIG. 5, the inclination of therespective straight line—except for lines 501 to 507—is similar. For theallocation of the image points to the world points e.g. Brute-Force orFast Library for Approximate Nearest Neighbors (FLANN)-based algorithmsmay be used. As it may basically be seen from parallel straight lines inFIG. 5, thus basically same and/or similar orientations of the recordingdevice in the real environment were determined for the several detectedfeatures of the object.

This becomes clearer from the histogram 600 illustrated in FIG. 6 inwhich the frequencies of the orientations of the recording device in thereal environment determined for the several detected features of theobject are plotted. The orientation is applied in the form of angle αindicating the rotation around the normal axis of the recording.Regarding the above discussed example with spatial directions X, Y, Zthe angle α thus corresponds to an orientation (alignment) of therecording device in the plane spanned by X and Z, i.e. a rotation aroundY. The frequency is plotted logarithmically.

It is evident from FIG. 6 that the angle α was determined for somedetected features of the object at approx. −65°, for some detectedfeatures of the object at approx. 60°, for some detected features of theobject at approx. 64°, for far more detected features of the object atapprox. 90° and for even more detected features of the object at approx.91°.

According to embodiments now the one of the orientations of therecording device in the real environment determined for the severaldetected features of the object which fulfills a quality criterion isdetermined to be the orientation of the recording device in the realenvironment. Relating to the example shown in FIG. 6, e.g. therespective one degree wide interval (bin) of the histogram with thegreatest number of entries may be selected. The quality criterion maythus e.g. be that the orientation of the recording device in the realenvironment is the most frequently determined orientation. Apart fromthat still further quality criteria may be used. It may for example berequested that the selected interval has to comprise a minimum number ofentries or that the selected bin has to represent at least apredetermined portion of the several detected features of the object(i.e. the bin has to represent the determined orientation each for thepredetermined portion of the several detected features of the object).

In the example shown in FIG. 6, the frequency for 90° and 91° dominantand absolute are in a similar range, so that both orientations mayfulfill a selected quality criterion. Accordingly, with neighboringand/or similar orientations (i.e. with neighboring bins or bins onlyseparated by a small number of bins in between) also the average valueof the two orientations may be determined as the orientation of therecording device in the real environment. Optionally, also a weightingof the neighboring and/or similar orientations may be executed (e.g.according to their frequency).

In the scope of the description of FIG. 7, in the following the secondapproach for determining the orientation of the recording device in thereal environment on the basis of determining a transformation whichcorrelates the at least one part of the recording of the known objectwith at least one part of a comparison recording is discussed.

In the second approach determining an orientation of the recordingdevice in the real environment comprises determining a comparisonrecording from the plurality of comparison recordings of a database.Determining a comparison recording from the plurality of comparisonrecordings of a database is here based on the known position of therecording device in the real environment. In other words: A comparisonrecording is selected from the database for which based on the positionof the recording device a high likelihood is given that it shows theobject at all and/or that is shows the object from the same or a similarperspective. An orientation at least of the selected comparisonrecording in the real environment is known here. In the database ofcourse also for each of the plurality of comparison recordings theorientation in the real environment may each be stored.

Furthermore, determining the orientation of the recording device in thereal environment comprises determining a rotation of the recording ofthe object relative to the comparison recording. I.e., an imageregistration of the recording of the object with respect to thecomparison recording is executed. For this purpose, known imageregistration methods may be used, like e.g. the Enhanced CorrelationCoefficient (ECC) algorithm. In this respect, the recording of theobject may e.g. be rotated stepwise with respect to the comparisonrecording, as indicated by the sequence of recordings 701 to 710 in FIG.7. In the recordings 701 to 710 each a roof window is illustrated as anexample for an object in the real environment in which the user movesaround. The recordings are here each rotated counter-clockwise from leftto right by 1° each. For each rotation the ECC algorithm determines acorrelation with the comparison image. Subsequently, the bestcorrelation is selected and a corresponding transformation matrix isdetermined. From the transformation matrix again the orientation, i.e.the rotation of the recording of the object relative to the comparisonrecording may be determined.

From the orientation of the comparison recording in the real environmentand the rotation of the recording of the object relative to thecomparison recording further the orientation of the recording device inthe real environment may be determined (by a combination of both piecesof information).

As indicated in FIG. 7, the second approach may e.g. exclusively be usedfor objects extending vertically above the user. For example, while therepresentation of the virtual environment is output to the user therecording device may be aligned vertically with respect to a straightdirection of view of the user in the real environment. In other words:The recording device in the real environment may be aligned heavenwardand/or in the direction of the ceiling of a room or a hall in which theuser moves. The object may accordingly e.g. either be an illuminationdevice, a (roof) window, a carrier, a beam at the ceiling of the roomand/or the hall. Accordingly, the plurality of comparison recordings inthe database may e.g. comprise different recordings of the ceiling ofthe room and/or hall. Alternatively, the recording device in the realenvironment may also be aligned in the direction of the floor of a roomor a hall in which the user moves. The object may then e.g. be a lightsource inserted into the floor (laser, LED) or a mark (e.g. an emergencydesignation) like e.g. an arrow (e.g. illuminated arrow). Generally, anobject according to the present disclosure may also be an especiallydesignated object, like e.g. an especially colored object (chromakeying).

If the representation of the virtual environment is again output to theuser via a display device mounted to the head of the user, the displaydevice may further comprise the recording device. The display device mayhere again comprise a mobile communications device including fixingdevice for mounting the mobile communications device to the head of theuser. By this, a camera of the mobile communications device may be usedas a recording device. Thus, a calibration of the representation of thevirtual environment may be enabled without additional hardwarecomponents. To be able to make recordings of the ceiling or the floor ofthe room and/or the hall using the camera of the mobile communicationsdevice, e.g. a periscope-like device may be used wherein one opening isdirected to the ceiling or the floor and wherein the other opening isdirected towards the lens of the camera of the mobile communicationsdevice. Via mirrors and/or prisms in the interior of the periscope-likedevice incident light beams may be deflected from the original directionof incidence (perpendicular to the first opening) towards the desiredoutput direction (perpendicular to the second opening).

To simplify image registration and thus reduce the necessary computingpower the plurality of comparison recordings may be binary recordings.Accordingly, determining the rotation of the recording of the objectrelative to the comparison recording comprises converting the recordingof the object into a binary recording of the object and determining therotation of the binary recording of the object relative to thecomparison recording. For determining the rotation of the binaryrecording of the object relative to the comparison recording again theabove mentioned image registration methods may be used.

Furthermore, the resolution of the plurality of comparison recordingsmay be limited (e.g. to 320×240 pixels) to save computing power.Accordingly, the method may comprise scaling the recording of theobject, i.e. the original resolution is scaled to a target resolution(e.g. from 1920×1080 pixels to 320×240 pixels). As indicated, the targetresolution may be lower than the original resolution. Due to the reducednumber of pixels in the recording of the object computing time may besaved.

Instead of comparing the complete recording to a reference recording,the orientation of the object in the recording may be determined andcompared to a reference direction (e.g. according to the methodsdescribed in context with FIGS. 9a to 9d ) to determine the orientationof the recording device in the real environment.

In the following, in FIG. 8 a method 800 for setting a direction of viewin a representation of a virtual environment according to a secondaspect of the present disclosure is illustrated.

Here, the method 800 comprises recording 802 a known object in a realenvironment using a recording device—as described hereinabove. I.e., therecording may e.g. be a still picture, a video or a sound recording.Accordingly, the recording device may comprise a still camera, a videocamera, a sound recording device or a combination thereof.

Further, the method 800 comprises determining 804 a rotational offset ofthe direction of view in the representation of the virtual environmentaround the yaw axis of the representation of the virtual environmentbased on the recording of the object and a current direction of view inthe representation of the virtual environment. The current direction ofview in the representation of the virtual environment may, for example,be received by a display device (e.g. HMD, HMU) which outputs therepresentation of the virtual environment to the user (and optionallycalculates the same) or by a computer which calculates the virtualenvironment (e.g. back end of a VR system).

The recording device may here—as described hereinabove—spatially belocated in close proximity to the user (e.g. at the head of the user).From the recording of the object the orientation of the recording devicein the real environment may be determined, which may then assumed to bethe approximate orientation of the head of the user in the realenvironment. Therefrom, using the information on the current directionof view in the representation of the virtual environment, the rotationaloffset of the direction of view in the representation of the virtualenvironment around the yaw axis of the representation of the virtualenvironment may be determined. Some examples for the determination ofthe orientation of the recording device in the real environment and alsofor the determination of the rotational offset of the direction of viewin the representation of the virtual environment are explained in moredetail in the following description.

The method 800 further comprises rotating 806 the direction of view inthe representation of the virtual environment by the rotational offset.In other words: The direction of view in the representation of thevirtual environment is corrected by a rotation around the yaw axis ofthe representation of the virtual environment, wherein the direction andthe magnitude of the rotation are determined by the rotational offset.The representation of the virtual environment is thus corrected by therotational offset. The representation of the virtual environment maythus be adapted to the actual position and orientation of the head ofthe user in the real environment. Among other things, also the method800 allows a calibration of the direction of view in the representationof the virtual environment. In particular, also using the method 800 anerroneously determined orientation in the real environment and/or adrifting of the direction of view in the representation of the virtualenvironment caused by measurement errors of the conventionally usedsensors for determining the position and the alignment (of the head) ofa user may be corrected.

Just like method 100 also method 800 may in some embodiments furthercomprise outputting the representation of the virtual environment to auser. Outputting the representation of the virtual environment to theuser may here e.g. be executed via a display device mounted to the headof the user which further comprises the recording device. In such anarrangement the orientation of the recording device in the realenvironment may be assumed to be approximately the orientation of thehead of the user in the real environment.

In some embodiments, the display device mounted to the head of the usercomprises a mobile communications device (e.g. a smartphone). Asindicated above, in a conventional operation of the VR system sensorsalready existing in the mobile communications device (e.g. gyroscope,magnetometer, accelerometer) may be used for determining the orientationof the head of the user in the real environment. By using the camera ofthe mobile communications device as a recording device, a rotationaloffset of the direction of view in the representation of the virtualenvironment around the yaw axis of the representation of the virtual dueto measurement errors of the sensors of the mobile communications devicemay be corrected. For the calibration of the representation of thevirtual environment, thus resources may be used already provided by themobile communications device. In other words: The method 800 may beexecuted directly (i.e. online) on the mobile communications device.Also the method 800 may thus enable a calibration of the representationof the virtual environment without additional hardware components.Alternatively, e.g. a part of the method 800 may be executed by themobile communications device and another part of the method 800, likee.g. determining 804 the rotational offset of the direction of view inthe representation of the virtual environment around the yaw axis of therepresentation of the virtual environment may be executed by an alreadyexisting back end of the VR system used by the user (i.e. offline). Thedetermined rotational offset may then e.g. be transmitted from theback-end to the mobile communications device so that the same may rotatethe direction of view in the current representation of the virtualenvironment by the rotational offset. The above described functionalitymay e.g. be implemented by an update for one or several already existingsoftware components of the VR system (e.g. software for the mobilecommunications device or software for the back end).

In some embodiments, determining 804 the rotational offset of thedirection of view in the representation of the virtual environmentaround the yaw axis of the representation of the virtual environment maycomprise determining an orientation of the recording device in the realenvironment based on the recording of the object and a referencedirection (exemplary methods will be discussed in the following). Thereference direction is a direction in the real environment whoseorientation relative to the object is known. In other words: Informationon the orientation of the object relative to the reference direction isused for determining the orientation of the recording device in the realenvironment. Further, determining 804 the rotational offset may comprisedetermining a target direction of view in the representation of thevirtual environment based on the orientation of the recording device inthe real environment. For example, the determined orientation of therecording device in the real environment may be provided to an algorithmfor the calculation of the representation of the virtual environmentwhich calculates a representation of the virtual environment basedthereon. In particular when the recording device is arranged at the headof the user, the target direction of view in the virtual environment maybe the direction of view in the virtual environment which corresponds tothe actual position and orientation of the head of the user in the realenvironment. For example, the recording device may be aligned verticallyto the straight ahead direction of view of the user in the realenvironment. The direction of view in the calculated representation ofthe virtual environment may consequently be regarded as the targetdirection of view.

From the target direction of view in the representation of the virtualenvironment and the current direction of view in the representation ofthe virtual environment, according to embodiments, now the rotationaloffset of the direction of view in the representation of the virtualenvironment around the yaw axis of the representation of the virtualenvironment is determined. This may, for example, be done by acomparison of the target direction of view and the current direction ofview in the representation of the virtual environment. In other words:It is determined to what extent the current direction of view in therepresentation of the virtual environment is rotated relative to thetarget direction of view in the representation of the virtualenvironment around the yaw axis of the representation of the virtualenvironment.

In the following FIGS. 9a to 9d exemplarily two different approaches fordetermining the orientation of the recording device in the realenvironment on the basis of an orientation of the object in therecording and a reference direction are explained.

For explaining the first approach in FIG. 9a a recording 900 of the roofof a hall is illustrated in which the user moves in the realenvironment. The recording 900 here shows a part of a longitudinalillumination device 910 representing an exemplary object. The objecthere is not restricted to longitudinal illumination devices. The objectmay e.g. also be a window, a carrier or a pattern at the ceiling of thehall and/or generally a room in the real environment in which the usermoves. The object may generally be an object extending exclusivelyvertically above the user. Accordingly, the recording device may beoriented vertically to a straight direction of view of the user in thereal environment, i.e. the recording device may be oriented heavenwardand/or towards the ceiling. Alternatively, the recording device in thereal environment may also be aligned in the direction of the floor of aroom or a hall in which the user moves. The object may then e.g. be alight source inserted into the floor (laser, LED) or a mark (e.g. anemergency designation) like e.g. an arrow (e.g. illuminated arrow). Ifoutputting the representation of the virtual environment to the user isexecuted via a display device mounted to the head of the user whichagain comprises a mobile communications device including fixing devicefor mounting the mobile communications device to the head of the user, acamera of the mobile communications device may be used as a recordingdevice. Thus, a calibration of the representation of the virtualenvironment may be enabled without additional hardware components. To beable to make recordings of the ceiling or the floor of the room and/orthe hall using the camera of the mobile communications device, e.g. aperiscope-like device may be used wherein one opening is directed to theceiling or the floor and wherein the other opening of the same isdirected towards the lens of the camera of the mobile communicationsdevice.

Determining the orientation of the recording device in the realenvironment, according to the first approach, comprises converting therecording of the object into a binary recording of the object. Thebinary recording 900′ corresponding to the recording 900 is illustratedin FIG. 9b . To generate the binary recording, optionally e.g. anenvironment-depending threshold value may be determined and/or definedfor the separation between the two possible states in the binaryrecording. Furthermore, the method comprises detecting candidates forthe object in the binary recording of the object. In the binaryrecording 900′ it is the area 910′ which corresponds to the longitudinalillumination device 910. In the binary recording 900′ only one candidate910′ for the object is illustrated, depending on the recording made,however, also two, three, four or more candidates for the object may bedetected in the recording and/or the corresponding binary recording. Inthis respect, optionally first the respective large and/or smallprincipal axis and also the gravity center of a candidate may bedetermined.

The method further comprises determining a respective (linear)eccentricity e of the candidates for the object. I.e., for each of thedetected candidates an eccentricity is determined. The determined lineareccentricity allows to estimate whether the possible candidate is arather circular (e≈0) or a rather longitudinal (e≈1) object. For thebinary recording 900′ thus the eccentricity of the area 910′ which isthe only candidate in the image, is determined. As the area 910′ islongitudinal, for the same a value of eccentricity of approximately oneis determined.

Further, the method comprises determining an orientation of a main axisof the one candidate whose eccentricity is above a threshold value andwhose main axis is longer than main axes of the other candidates for theobject with an eccentricity above the threshold value as an orientationof the object in the recording. For all candidates thus their determinedeccentricity is compared to a threshold value to determine thecandidates which represent a longitudinal object. For example, thethreshold value may thus be 0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85or 0.9. Out of the remaining candidates for the object the one with thelongest main axis is selected. The orientation of this candidate in therecording is determined to be the orientation of the object in therecording.

The orientation of the candidate in the recording may e.g. be determinedon the basis of an auxiliary vector 920, the auxiliary vector 920indicating the straight direction of view of the user. Defining theauxiliary vector 920 to be in the straight direction of view of the usermay enable to regard the orientation of the recording device as beingbasically identical to the orientation of the user in the realenvironment. Accordingly, from the determined orientation of therecording device the respective direction of view in the virtualenvironment may be determined as the target direction of view in therepresentation of the virtual environment which corresponds to theactual position and orientation of the head of the user in the realenvironment. For the area 910′ it may thus be determined that theorientation 930 of its main axis spans an angle of 89° with respect tothe auxiliary vector 920. I.e., the main axis of the area 910′ isrotated by 89° with respect to the auxiliary vector. Thus, theorientation 930 of the main axis of the area 910′ is determined to bethe orientation of the object in the recording.

Together with the information on the reference direction, from theorientation of the object in the recording the orientation of therecording device may be determined. As indicated above, the referencedirection is a direction in the real environment whose orientationrelative to the object is known. In other words: The orientation(alignment) of the object relative to the reference direction is known.The reference direction may e.g. be determined for a known environmentor be determined from reference recordings. If the user moves in thereal environment, e.g. within a hall with a basically rectangularfootprint, a corner of the footprint may be determined to be the origin.Starting from the origin (analog to the example shown in FIG. 2) threeorthogonal spatial axes X, Y and Z may be defined. From the definedoriginal point in the corner of the footprint X may e.g. point to theright (i.e. basically along a first boundary of the footprint), Y maypoint upwards (i.e. heavenward) (i.e. be basically perpendicular to thefootprint) and Z may point into the depth (i.e. to the front) (i.e.basically along a second boundary of the footprint which is orthogonalto the first boundary of the footprint). The unity direction Y thuscorresponds to the yaw axis, i.e. a rotation around this axishorizontally shifts a recording. The spatial axis Z may then e.g. beselected to be the reference direction which passes basically along thesecond boundary of the footprint. The orientation of the object—in theexample shown in FIG. 9a thus the longitudinal illumination device910—relative to the reference direction is known. For example, thelongitudinal illumination device 910 may be orthogonal to the referencedirection Z, i.e. parallel to the spatial direction X.

From the determined orientation of the object in the recording and theknown orientation of the object relative to the reference direction nowthe orientation of the recording device may be determined. In the aboveexample, thus the orientation of the auxiliary vector 920 in the realenvironment is determined. Thus, an orientation of the recording devicein the plane spanned by spatial directions Y and Z may be determined.

To save computing power, the resolution of the recording to be assessedmay be limited (e.g. to 320×240 pixels). Accordingly, the method maycomprise scaling the recording of the object, i.e. the originalresolution is scaled to a target resolution (e.g. from 1920×1080 pixelsto 320×240 pixels). As indicated, the target resolution may be lowerthan the original resolution. Due to the reduced number of pixels in therecording of the object computing time may be saved.

Within the scope of the description of FIGS. 9c and 9d , in thefollowing the second approach for determining the orientation of therecording device in the real environment on the basis of an orientationof the object in the recording and a reference direction are explained.

For explaining the second approach, in FIG. 9c a recording 940 of theroof of a hall is illustrated in which the user moves in the realenvironment. The recording 900 here shows a (linear) arrangement ofcircular illumination devices 951, 952, 951 representing an exemplaryobject. The object here is not restricted to an arrangement of circularillumination devices. The object may generally be any arrangement ofcircular objects at the ceiling of the hall and/or generally of a roomin the real environment in which the user moves. The object maygenerally be an object extending exclusively vertically above the user.Alternatively, the recording device in the real environment may also bealigned in the direction of the floor of a room or a hall in which theuser moves. The object may then e.g. be a light source inserted into thefloor (laser, LED) or a mark, like e.g. an arrow. Accordingly, therecording device may be oriented vertically to a straight direction ofview of the user in the real environment, i.e. the recording device maybe oriented heavenward (i.e. to the ceiling) and/or towards the floor.If outputting the representation of the virtual environment to the useris executed via a display device mounted to the head of the user whichagain comprises a mobile communications device including fixing devicefor mounting the mobile communications device to the head of the user, acamera of the mobile communications device may be used as the recordingdevice. Thus, a calibration of the representation of the virtualenvironment may be enabled without additional hardware components. To beable to make recordings of the ceiling or the floor of the room and/orthe hall using the camera of the mobile communications device, e.g. aperiscope-like device may be used wherein one opening of the same isdirected to the ceiling or the floor and wherein the other opening isdirected towards the lens of the camera of the mobile communicationsdevice.

Determining the orientation of the recording device in the realenvironment, according to the second approach, again comprisesconverting the recording of the object into a binary recording of theobject. The binary recording 940′ corresponding to the recording 940 isillustrated in FIG. 9d . To generate the binary recording, optionallye.g. an environment-depending threshold value may be determined and/ordefined for the separation between the two possible states in the binaryrecording.

Furthermore, the method comprises detecting circular objects in thebinary recording of the object. Respective radii of the circular objectsare here comprised in a predetermined value range. In other words: Onlycircular objects are detected whose value for the radius is greater thana first threshold value and smaller than a second threshold value. Thethreshold values may here be selected on the basis of information on thereal environment in which the user is moving (e.g. height of the roofand/or distance of the illumination devices from the floor, dimensionsof the illumination devices). For detecting circular objects in thebinary recording e.g. a Circular Hough Transfrom (CHT) based algorithmmay be used. Accordingly, in the binary recording the circular objects951′, 952′ and 953′ are detected which correspond to the arrangement ofcircular illumination devices 951, 952, 953 in the recording 940. Thelight areas 952″ and 953″ in the binary recording 940′, however,adjacent to the areas 952′ and 953′ are no circular objects as they donot fulfill the radius criterion. The light areas 952″ and 953″ in thebinary recording 940′ adjacent to the areas 952′ and 953′ correspond tothe optical effects 954, 955 in the recording 940 and thus do notrepresent objects in the real environment. Thus, with the help of theradius criterion optical interference and/or disruptive effects may beexcluded from the further method of determining the orientation of therecording device.

Further, the method comprises determining distances of the circularobjects from one another. In this respect, e.g. the center points of thecircular objects 951′, 952′ and 953′ may be determined and the distancesof the center points to each other may be determined. Also the radii ofthe respective circular objects may be comprised in the distancedetermination.

Additionally, determining the orientation of the recording device in thereal environment according to the second approach comprises determiningthe orientation of the object in the recording on the basis of thedistances of the circular objects from one another. From the distancesof the circular objects a relation between the individual circularobjects may be determined. Here, e.g. again information on the realenvironment in which the user is moving may be used. In the exampleshown in FIGS. 9c and 9d , e.g. the distances between the individualillumination devices in the linear arrangement of circular illuminationdevices 951, 952, 953 in the real environment and the distance of thelinear arrangement of circular illumination devices 951, 952, 953 to afurther linear arrangement of circular illumination devices (not shownin FIG. 9c ) may be used. In general, information on the geometry andcondition of the area in the real environment (e.g. ceiling of a room ora hall) detectable by the recording device may be included in thedetermination of the relation of the individual objects to each other.

In FIG. 9d it is determined for the distances of the circular objects951′, 952′ and 953′ to each other that the same correspond to thedistances of a linear arrangement of illumination devices in the realenvironment and that the circular objects 951′, 952′ and 953′ thusrepresent a known object in the real environment. Accordingly, from therespective positions of the circular objects 951′, 952′ and 953′ in thebinary recording a directional vector 970 of the object in the recordingrepresented by the circular objects 951′, 952′ and 953′ is determined.In this respect, e.g. a straight line may be fitted to the center pointsof the circular objects 951′, 952′ and 953′.

The orientation of the directional vector 970 (i.e. the object) in therecording may e.g. be determined on the basis of an auxiliary vector 960indicating e.g. the straight direction of view of the user. For thedirectional vector 970 it may thus be determined that the orientation ofthe object spans an angle of 35° with respect to the auxiliary vector960. I.e., the linear arrangement of circular illumination devices 951,952, 953 represented by the directional vector 970 is rotated by 35°with respect to the auxiliary vector 960. Thus, the orientation of thedirectional vector 970 is determined as an orientation of the object inthe recording.

Together with the information on the reference direction, from theorientation of the object in the recording the orientation of therecording device may be determined. As indicated above, the referencedirection is a direction in the real environment whose orientationrelative to the object is known. With reference to the coordinate systemXYZ exemplarily introduced in the description of FIG. 9b , as areference direction e.g. again the spatial axis Z may be selected whichpasses basically along the second boundary of the footprint. Theorientation of the object—in the example shown in FIG. 9c thus thelinear arrangement of circular illumination devices 951, 952,951—relative to the reference direction is known. For example, thelinear arrangement of circular illumination devices 951, 952, 951 may beorthogonal to the reference direction Z, i.e. parallel to the spatialdirection X.

From the determined orientation of the object in the recording and theknown orientation of the object relative to the reference direction nowthe orientation of the recording device may be determined. In the aboveexample, thus the orientation of the auxiliary vector 960 in the realenvironment is determined. Thus, an orientation of the recording devicein the plane spanned by spatial directions Y and Z may be determined.

To save computing power, the resolution of the recording to be assessedmay be limited (e.g. to 320×240 pixels). Accordingly, the method maycomprise scaling the recording of the object, i.e. the originalresolution is scaled to a target resolution (e.g. from 1920×1080 pixelsto 320×240 pixels). As indicated, the target resolution may be lowerthan the original resolution. Due to the reduced number of pixels in therecording of the object computing time may be saved.

In the approach described in context with FIGS. 9c and 9d it was assumedthat the recording device makes recordings in a plane substantiallyorthogonal to a plane in which the user is moving. For example, that theuser is moving in a hall and the recording device is making recordingsof the ceiling of the hall at an angle of basically 90° to the same.However, the recording device may also be tilted with respect to theceiling (e.g. when the recording device is mounted to the head of theuser and the same makes a nodding or tilting movement with his head). Tobe able to correctly determine the distances of the circular objects inthe binary recording also in such situations, additionally apart from aknown position of the recording device in the real environment (e.g.position of the user in the real environment determined by the VRsystem) also actual measurement values e.g. of a gyroscope and/or anaccelerometer of a mobile communications device used as a display devicefor the virtual environment and as a recording device may be utilized.

Current measurement values of a gyroscope and/or an accelerometer of amobile communications device used as a display device for the virtualenvironment and as a recording device may generally also be used (i.e.in all embodiments of the present disclosure) to determine whether themoment is suitable to make a recording. For example, it may bedetermined that the recording device makes recordings only in a certainvalue range of the measurement values. It may thus be prevented, thatrecordings are assessed which comprise blurs or other image distortionsdue to the orientation of the recording device in the real environment.It may thus be avoided that an erroneous orientation of the recordingdevice in the real environment is determined and thus the direction ofview in the virtual environment is rotated by an erroneous rotationaloffset.

In some embodiments, via the display device mounted to the head of theuser which comprises the recording device a recording of a light sourcemounted to the body of the user (e.g. waist, trouser waistband) is made.For example, a laser may be arranged at the torso of the user whichemits a laser beam in the direction of a straight ahead movement of theuser (i.e. basically along a straight direction of view of the user).Thus, the light source and/or the laser beam is the known object whichis recorded by the recording device. From the current position of therecording device and the known position of the recording device at atleast one previous point in time an orientation of the body in the realenvironment is determined (i.e. the motion vector at the time instant ofthe recording is assumed to be the orientation of the body). In otherwords: The orientation of the body in the real environment serves as areference direction. From the orientation of the laser beam in therecording now the orientation of the recording device relative to thelaser beam is determined. The orientation of the laser beam in therecording may e.g. be determined according to the method described incontext with FIGS. 9a and 9b . As the direction of the laser beamcorresponds to the reference direction, the absolute orientation of therecording device in the real environment may be determined from therecording of the laser beam. Accordingly, from the absolute orientationin the real environment a target direction of view in the representationof the virtual environment may be determined so that by a comparisonwith the current direction of view in the representation of the virtualenvironment again the rotational offset of the direction of view in therepresentation of the virtual environment around the yaw axis of therepresentation of the virtual environment may be determined. Thedirection of view in the representation of the virtual environment maythen be rotated, i.e. corrected, by the rotational offset.

FIG. 10 shows a further method 1000 for setting a direction of view in arepresentation of a virtual environment.

The method 1000 here comprises recording 1002 an object arranged in areal environment at the body of the user using a recording devicearranged at the head of the user at a first time instant t₀ and at alater second time instant t₁. The object may e.g. be of a light sourcemounted to the body of the user (e.g. waist, trouser waistband). Forexample, a laser may be arranged at the torso of the user which emits alaser beam in the direction of a straight ahead movement of the user(i.e. basically along a straight direction of view of the user).

A user may move his head by rotation around a transverse axis of thehead, rotation around a longitudinal axis of the head, and/or rotationaround a yaw axis of the head. The transverse axis, the longitudinalaxis and the yaw axis of the head here are perpendicular to each other.

As the recording device is arranged at the head of the user, also thesame is movable around the pitch/transverse axis, the roll/longitudinalaxis and the yaw axis of the head.

As described hereinabove, the representation of the virtual environmentmay be output to the user via a display device mounted to the head ofthe user. In a conventional operation of the VR system then e.g. sensors(e.g. gyroscope, magnetometer, accelerometer) already existing in themobile communications device (HMD) may be used for determining theorientation of the head of the user in the real environment. Thus, inparticular a rotary position of the head around its yaw axis may bedetermined. However, determining the orientation of the head using theexisting sensors—as illustrated above—is defective. Apart from thesensors above, i.e. at least one further sensor, the display device mayfurther comprise the recording device. Thus, the method 1000 furthercomprises determining 1004 a rotational offset of the direction of viewin the representation of the virtual environment around a yaw axis ofthe representation of the virtual environment based on the recordings ofthe object at the first time instant t₀ and a second time instant t₁ andmeasurement values of at least one further sensor mounted to the head ofthe user.

The rotational offset of the direction of view in the representation ofthe virtual environment around a yaw axis of the representation of thevirtual environment here corresponds to a rotational offset of therecording device around the yaw axis of the head of the user. The sameis determined by a comparison of the rotation of the recording devicearound a yaw axis of the head of the user determined from recordings ofthe object at time instants t₀ and t₁ and the rotation of the recordingdevice around a yaw axis of the head of the user determined from themeasurement values between the time instants t₀ and t₁ from the sensormounted to the head of the user. Thus, the rotational offset of thedirection of view in the representation of the virtual environmentaround the yaw axis of the representation of the virtual may bedetermined without having to determine the absolute orientation of therecording device and/or the head of the user in the real environment.Rather, it is sufficient, to determine the relative rotational offset ofthe recording device and/or the HMD around the yaw axis of the head.

The method 1000 thus further comprises rotating 1006 the direction ofview in the representation of the virtual environment by the rotationaloffset. In other words: The direction of view in the representation ofthe virtual environment is corrected by a rotation around the yaw axisof the representation of the virtual environment, wherein the directionand the magnitude of the rotation are determined by the rotationaloffset. The representation of the virtual environment is thus correctedby the rotational offset. The representation of the virtual environmentmay thus be adapted to the actual position and orientation of the headof the user in the real environment. Among other things, also the method1000 allows a calibration of the direction of view in the representationof the virtual environment. In particular, also using the method 1000 anerroneously determined orientation in the real environment and/or adrifting of the direction of view in the representation of the virtualenvironment caused by measurement errors of the conventionally usedsensors for determining the position and the alignment (of the head) ofa user may be corrected.

According to some embodiments, determining 1004 the rotational offset ofthe direction of view in the representation of the virtual environmentaround the yaw axis of the representation of the virtual environmentincudes determining a first rotation of the recording device around theyaw axis of the head of the user between the first time instant t₀ andthe second time instant t₁ based on the recordings of the object at thefirst time instant t₀ and at the second time instant t₁. Here, for thefirst time instant t₀ the orientation of the recording device relativeto the laser beam is determined from the recording of the laser beam atthe first time instant t₀. For example, the laser beam may be directedin the direction of a straight ahead movement of the user and the useris looking straight ahead, so that a rotation of the recording devicearound the yaw axis of the head by 0° relative to the laser beam isdetermined to be a first orientation. For the second time instant t₁ theorientation of the recording device relative to the laser beam isdetermined from the recording of the laser beam at the second timeinstant t₁. If the user has his head turned to the side at the timeinstant t₁, a second orientation of the laser beam different from thefirst orientation in the recording is determined, i.e. a rotation of therecording device around the yaw axis of the head relative to the laserbeam different from 0°. The orientation of the laser beam in therecording may e.g. be determined according to the method described incontext with FIGS. 9a and 9b . Thus, the relative rotational anglearound the yaw axis of the head between the orientation of the head at afirst time instant t₀ and the orientation of the head at the second timeinstant t₁ is determined from the two recordings.

Further, determining 1004 the rotational offset of the direction of viewin the representation of the virtual environment around the yaw axis ofthe representation of the virtual environment in these embodimentsincudes determining a second rotation of the recording device around theyaw axis of the head of the user between the first time instant t₀ andthe second time instant t₁ based on the measurement values of at leastone further sensor mounted to the head of the user. In other words: Fromthe measurement values of at least one further sensor mounted to thehead of the user between the time instants t₀ and t₁ again an effective(i.e. total) rotation around the yaw axis of the head is determined as acomparison value. Thus, the relative rotational angle around the yawaxis of the head between the orientation of the head at a first timeinstant t₀ and the orientation of the head at the second time instant t₁is determined from the measurement values. Due to measurement errors ofthe sensors of the display device, however, the rotation around the yawaxis of the head determined from the measurement values may beerroneous. As the same is used in the VR system for the determination ofthe direction of view in the representation of the virtual environment,also the direction of view in the representation of the virtualenvironment may be erroneous, i.e. rotated around the yaw axis of therepresentation of the virtual environment.

Thus, determining 1004 the rotational offset of the direction of view inthe representation of the virtual environment around the yaw axis of therepresentation of the virtual environment in this embodiment furthercomprises determining a rotational offset around the yaw axis of thehead between the first rotation and the second rotation as a rotationaloffset of the direction of view in the representation of the virtualenvironment around the yaw axis of the representation of the virtualenvironment. In other words: The rotational offset around the yaw axisof the head between the first rotation and the second rotationrepresents the difference (i.e. the variation) between the firstrotation determined from the recordings of the object and the secondrotation determined from the measurement values of at least one furthersensor. The resulting rotational offset around the yaw axis of the headof the user due to the measurement errors of the sensors of the displaydevice (i.e. the HMD) is thus determined using the exact determinationof the rotation of the display device relative to the object at the bodyof the user (e.g. the laser beam). The relative rotational offset of therecording device and/or the HMD around the yaw axis of the head may thusbe assumed to be the rotational offset of the direction of view in therepresentation of the virtual environment around the yaw axis of therepresentation of the virtual environment, so that the same mayaccordingly be corrected by the rotational offset of the recordingdevice and/or the HMD around the yaw axis of the head.

In the following FIGS. 11a and 11b exemplary connections between amotion vector of a user 1000 in the real environment, an actualdirection of view v_(real) of the user 1100 in the real environment anda direction of view v_(vr) in the representation of the virtualenvironment determined from the measurement values of at least onefurther sensor arranged at the head 1102 of the user 1100. In the FIGS.11a and 11b the user 1100 moves along the motion vector p between thefirst time instant t₀ and the later second time instant t₁, the vectorbeing determinable e.g. by position measurements m(t₀) and m(t₁) at thetwo time instants t₀ and t₁.

FIG. 11a here shows the situation at the first time instant t₀. At thistime instant the user 1100 is looking straight ahead, i.e. in thedirection of its motion vector p. The actual orientation of the head ofthe user is determined for the first time instant t₀ by means of arecording device included in a HMD 1104 arranged at the head 1102 of theuser 1100. As indicated above, using the recording device a firstrecording of an object (e.g. light source; laser light source emittinglaser beams) arranged at the body 1106 of the user 1100 is made andtherefrom a relative orientation of the head with respect to the objectis determined for the first time instant t₀. The alignment of the objectrelative to the body 1106 (e.g. torso) of the user 1100 is known here.For example, a laser beam may be directed in the direction of a straightahead movement of the user 1100, so that a rotation of the recordingdevice around the yaw axis of the head by 0° relative to the laser beamis determined as a first orientation. As the laser beam is directed inthe direction of the straight ahead movement of the user 1100, thedirection of the laser beam basically corresponds to the motion vector pof the user 1100 so that also the rotation θ of the recording devicearound the yaw axis of the head is known relative to the motion vector pof the user 1100, here a rotation by 0°. As the motion vector is known,thus the absolute orientation of the recording device and thus also ofthe head 1102 of the user 1100 in the real environment is known.

In the usual operation of the VR system and by means of at least onefurther sensor included in the HMD 1104 (e.g. gyroscope, magnetometer,accelerometer) also the orientation of the head 1102 of the user 1100 inthe real environment and therefrom the direction of view v_(vr) in therepresentation of the virtual environment is determined. For the firsttime instant t₀ now the direction of view of the user 1100 in the realenvironment determined from the measurement values of at least onefurther sensor arranged at the head 1102 of the user 1100 is assumed tobe identical to the actual direction of view v_(real) of the user 1100in the real environment. Thus, also the direction of view v_(vr) in therepresentation of the virtual environment would basically correspond tothe motion vector p of the user 1100 so that between the actualdirection of view v_(real) of the user 1100 in the real environment andthe direction of view v_(vr) in the representation of the virtualenvironment a rotation of basically 0° may be assumed. In other words:The rotational offset around the yaw axis of the representation of thevirtual is basically 0°.

In FIG. 11b the situation at the second time instant t₁ is shown. At thesecond time instant t₁ the user 1100 is looking straight ahead, i.e.basically in the direction of its motion vector p. From the secondrecording of the object (e.g. laser) arranged at the body 1106 of theuser at the second time instant t₁ thus again a rotation of therecording device around the yaw axis of the head of 0° relative e.g. tothe laser beam is determined to be the second orientation, i.e., therotation θ of the recording device around the yaw axis of the headrelative to the motion vector p of the user 1100 is again 0°. For thesecond time instant t₁ now the direction of view of the user 1100 in thereal environment determined from the measurement values of at least onefurther sensor arranged at the head 1102 of the user 1100 is notidentical to the actual direction of view v_(real) of the user 1100 inthe real environment determined from the second recording due tomeasurement errors. Thus, the direction of view v_(v)r in therepresentation of the virtual environment erroneously does basically notcorrespond to the motion vector p of the user 1100, so that between theactual direction of view v_(real) of the user 1100 in the realenvironment and the direction of view v_(v)r in the representation ofthe virtual environment a rotation ε different from 0° is present. Inother words: The rotational offset around the yaw axis of therepresentation of the virtual environment is different to 0°. Theconsequence of the rotational offset different to 0° around the yaw axisof the representation of the virtual environment is that the user doesnot basically move along the motion vector p in the virtual environmentbut at an inclined angle to the same. If the user were e.g. walkingstraight ahead in the real environment, he would be moving at an angleto the front in the real environment.

In order to now correct the direction of view in the representation ofthe virtual environment, as illustrated above (see description of FIG.11) the relative rotational offset around the yaw axis of the head 1102of the user 1100 between the rotation determined from the two recordingsand the rotation determined from the measurement values of at least onefurther sensor may be used. Alternatively, the absolute orientation ofthe recording device (and thus also of the head 1102) in the realenvironment determinable due to the knowledge of the relativeorientation of the recording device to the known motion vector may beused to determine a target direction of view in the representation ofthe virtual environment and to correct the direction of view in therepresentation of the virtual environment by the rotational offsetbetween the target direction of view and the current direction of viewin the representation of the virtual environment (i.e. rotate around theyaw axis of the representation of the virtual environment). Thus, acalibration of the direction of view in the representation of thevirtual environment may be enabled.

The features disclosed in the above description, the enclosed claims andthe enclosed Figures may both individually and in any combination be ofimportance and implemented for realizing an embodiment in their variousforms.

Although some aspects have been described in connection with anapparatus, it is clear that these aspects also illustrate a descriptionof the corresponding method, where a block or a device of an apparatusis to be understood as a method step or a feature of a method step.Analogously, aspects described in the context of or as a method stepalso represent a description of a corresponding block or detail orfeature of a corresponding apparatus.

Depending on certain implementation requirements, embodiments of theinvention can be implemented in hardware or in software. Theimplementation can be performed using a digital storage medium, forexample a floppy disk, a DVD, a Blue-Ray, a CD, a ROM, a PROM, an EPROM,an EEPROM or a FLASH memory, a hard disc or another magnetic or opticalmemory having electronically readable control signals stored thereon,which cooperate or are capable of cooperating with a programmablehardware component such that the respective method is performed.

A programmable hardware component may be formed by a processor, aCentral Processing Unit (CPU), a Graphics Processing Unit (GPU), acomputer, a computer system, an Application-Specific Integrated Circuit(ASIC), an Integrated Circuit (IC), a System on Chip (SOC), aprogrammable logics element or a Field Programmable Gate Array (FPGA)comprising a microprocessor.

Therefore, the digital storage medium may be machine or computerreadable. Some embodiments comprise also a data carrier comprisingelectronically readable control signals which are capable of cooperatingwith a programmable computer system or a programmable hardware componentsuch that one of the methods described herein is performed. Oneembodiment is thus a data carrier (or a digital storage medium or acomputer readable medium) on which the program for executing of themethods described herein is stored.

Generally speaking, embodiments of the present invention may beimplemented as a program, firmware, a computer program or a computerprogram product having a program code or as data, wherein the programcode or the data is effective to execute one of the methods when theprogram is executed on a processor, or a programmable hardwarecomponent. The program code or the data may, for example, also be storedon a machine-readable carrier or data carrier. The program code or thedata may among others be present as a source code, machine code or bytecode or any other intermediate code.

A further embodiment is a data stream, a signal sequence or a sequenceof signals which may represent the program for executing one of themethods described herein. The data stream, the signal sequence or thesequence of signals may for example be configured so as to betransferred via a data communication connection, for example via theinternet or another network. Embodiments thus also are signal sequencesrepresenting data suitable for being transferred via a network or a datacommunication connection, the data representing the program.

A program according to one embodiment may implement one of the methodsduring its execution for example by reading out memory locations orwriting one or several data into the same, whereby possibly switchingprocesses or other processes in transistor structures, in amplifierstructures or in other electrical, optical, magnetic or other membersoperating according to another functional principle are caused.Accordingly, by reading out a memory location, data, values, sensorvalues or other information is determined, detected or measured by aprogram. By reading out one or several memory locations, a program maydetect, determine or measure magnitudes, values, measured quantities andother information and, by writing into one or several memory locations,cause, trigger or execute an action and control other devices, machinesand components.

The above described embodiments are merely an illustration of theprinciples of the present invention. It is understood that modificationsand variations of the arrangements and the details described herein willbe apparent to others skilled in the art. It is the intent, therefore,that this invention is limited only by the scope of the impending patentclaims and not by the specific details presented by way of descriptionand explanation of the embodiments herein.

1. A method for setting a direction of view in a representation of avirtual environment, comprising: recording a known object in a realenvironment using a recording device; determining a rotational offset ofthe direction of view in the representation of the virtual environmentaround a yaw axis of the representation of the virtual environment basedon the recording of the object, a known position of the recording devicein the real environment and a current direction of view in therepresentation of the virtual environment for a time instant t₀; androtating the direction of view in the representation of the virtualenvironment by the rotational offset, wherein the current direction ofview in the representation of the virtual environment for the timeinstant t₀ is based on measurement values of a gyroscope, a magnetometerand/or an accelerometer, and wherein determining the rotational offsetof the direction of view in the representation of the virtualenvironment around the yaw axis of the representation of the virtualenvironment comprises: determining an orientation of the recordingdevice in the real environment based on the recording of the object andthe known position of the recording device in the real environment;determining a target direction of view in the representation of thevirtual environment for the time instant t₀ based on the orientation ofthe recording device in the real environment; and determining therotational offset of the direction of view in the representation of thevirtual environment around the yaw axis of the representation of thevirtual environment from the target direction of view in therepresentation of the virtual environment for the time instant t₀ andthe current direction of view in the representation of the virtualenvironment for the time instant t₀.
 2. (canceled)
 3. The methodaccording to claim 1, wherein determining the orientation of therecording device in the real environment comprises determining atransformation which correlates at least a part of the recording of theobject with at least a part of a comparison recording.
 4. The methodaccording to claim 1, wherein determining the orientation of therecording device in the real environment further comprises: detecting afeature of the object in the recording; determining a position of thefeature in the recording; identifying a comparison feature from aplurality of comparison features of a database which corresponds to thefeature of the object in the recording, wherein a position in the realenvironment is associated with each of the plurality of comparisonfeatures; and determining the orientation of the recording device in thereal environment based on the known position of the recording device inthe real environment, the position of the feature in the recording andthe position in the real environment associated with the identifiedcomparison feature.
 5. The method according to claim 4, wherein whendetermining the orientation of the recording device in the realenvironment several features of the object are detected, wherein anorientation of the recording device in the real environment isdetermined for each of the several detected features of the object, andwherein the one of the orientations of the recording device in the realenvironment determined for the several detected features of the objectwhich fulfils a quality criterion is determined to be the orientation ofthe recording device in the real environment.
 6. The method according toclaim 4, wherein the plurality of comparison features comprisesdifferent features of the object.
 7. The method according to claim 1,wherein the object is a pattern attached to a vertical plane.
 8. Themethod according to claim 1, wherein the method further comprisesoutputting a representation of the virtual environment to a user andwherein the recording device is aligned in the straight direction ofview of the user in the real environment.
 9. The method according toclaim 1, wherein determining the orientation of the recording device inthe real environment comprises: determining, based on the known positionof the recording device in the real environment, a comparison recordingfrom a plurality of comparison recordings of a database, wherein anorientation of the comparison recording in the real environment isknown; determining a rotation of the recording of the object relative tothe comparison recording; determining the orientation of the recordingdevice in the real environment based on the orientation of thecomparison recording in the real environment and the rotation of therecording of the object relative to the comparison recording.
 10. Themethod according to claim 9, wherein the plurality of comparisonrecordings are binary recordings and wherein determining the rotation ofthe recording of the object relative to the comparison recordingcomprises: converting the recording of the object into a binaryrecording of the object; and determining the rotation of the binaryrecording of the object relative to the comparison recording.
 11. Themethod according to claim 9, wherein the method further comprisesoutputting the representation of the virtual environment to a user andwherein the recording device is aligned vertically to a straightdirection of view of the user in the real environment.
 12. The methodaccording to claim 11, wherein the object extends vertically exclusivelyabove or exclusively below the user.
 13. The method according to claim8, wherein outputting the representation of the virtual environment tothe user is executed via a display device mounted to the head of theuser and wherein the display device further comprises the recordingdevice.
 14. The method according to claim 13, wherein the display devicecomprises a mobile communications device and wherein the recordingdevice is a camera of the mobile communication device.
 15. A method forsetting a direction of view in a representation of a virtualenvironment, comprising: recording a known object in a real environmentusing a recording device; determining a rotational offset of thedirection of view in the representation of the virtual environmentaround a yaw axis of the representation of the virtual environment basedon the recording of the object and a current direction of view in therepresentation of the virtual environment for a time instant t₀; androtating the direction of view in the representation of the virtualenvironment by the rotational offset, wherein the current direction ofview in the representation of the virtual environment for the timeinstant t₀ is based on measurement values of a gyroscope, a magnetometerand/or an accelerometer, and wherein determining the rotational offsetof the direction of view in the representation of the virtualenvironment around the yaw axis of the representation of the virtualenvironment comprises: determining an orientation of the recordingdevice in the real environment based on an orientation of the object inthe recording and a reference direction; determining a target directionof view in the representation of the virtual environment for the timeinstant t₀ based on the orientation of the recording device in the realenvironment; and determining the rotational offset of the direction ofview in the representation of the virtual environment around the yawaxis of the representation of the virtual environment from the targetdirection of view in the representation of the virtual environment forthe time instant t₀ and the current direction of view in therepresentation of the virtual environment for the time instant t₀. 16.(canceled)
 17. The method according to claim 15, wherein determining theorientation of the recording device in the real environment comprises:converting the recording of the object into a binary recording of theobject; detecting candidates for the object in the binary recording ofthe object; determining a respective eccentricity of the candidates forthe object; and determining an orientation of a main axis of the onecandidate whose eccentricity is above a threshold value and whose mainaxis is longer than main axes of the other candidates for the objectwith an eccentricity above the threshold value as an orientation of theobject in the recording.
 18. The method according to claim 15, whereindetermining the orientation of the recording device in the realenvironment further comprises: converting the recording of the objectinto a binary recording of the object; detecting circular objects in thebinary recording of the object, wherein respective radii of the circularobjects are included in a predetermined value range; determiningdistances of the circular objects from one another; determining theorientation of the object in the recording on the basis of the distancesof the circular objects from one another.
 19. The method according toclaim 15, wherein the method further comprises outputting therepresentation of the virtual environment to a user and wherein therecording device is aligned vertically to a straight direction of viewof the user in the real environment.
 20. The method according to claim19, wherein the object extends vertically exclusively above orexclusively below the user.
 21. The method according to claim 19,wherein outputting the representation of the virtual environment to theuser is executed via a display device mounted to the head of the userand wherein the display device further comprises the recording device.22. The method according to claim 21, wherein the display devicecomprises a mobile communications device and wherein the recordingdevice is a camera of the mobile communications device.
 23. A method forsetting a direction of view in a representation of a virtualenvironment, comprising: recording an object arranged in a realenvironment at the body of a user using a recording device arranged atthe head of the user at a first time instant and at a later second timeinstant; determining a rotational offset of the direction of view in therepresentation of the virtual environment around a yaw axis of therepresentation of the virtual environment based on the recordings of theobject at the first time instant and at the second time instant andmeasurement values of at least one further sensor mounted to the head ofthe user; and rotating the direction of view in the representation ofthe virtual environment by the rotational offset.
 24. The methodaccording to claim 23, wherein determining the rotational offset of thedirection of view in the representation of the virtual environmentaround the yaw axis of the representation of the virtual environmentfurther comprises: determining a first rotation of the recording devicearound a yaw axis of the head of the user between the first time instantand the second time instant based on the recordings of the object at thefirst time instant and at the second time instant; determining a secondrotation of the recording device around the yaw axis of the head of theuser between the first time instant and the second time instant based onthe measurement values of at least one further sensor mounted to thehead of the user; and determining the rotational offset between thefirst rotation and the second rotation as the rotational offset of thedirection of view in the representation of the virtual environmentaround the yaw axis of the representation of the virtual environment.25. The method according to claim 23, wherein the method furthercomprises outputting the representation of the virtual environment tothe user via a display device mounted to the head of the user andwherein the display device further comprises the recording device andthe at least one further sensor.